Pairwise-Covariance Linear Discriminant Analysis

In machine learning, linear discriminant analysis (LDA) is a popular dimension reduction method. In this paper, we first provide a new perspective of LDA from an information theory perspective. From this new perspective, we propose a new formulation of LDA, which uses the pairwise averaged class covariance instead of the globally averaged class covariance used in standard LDA. This pairwise (averaged) covariance describes data distribution more accurately. The new perspective also provides a natural way to properly weigh different pairwise distances, which emphasizes the pairs of class with small distances, and this leads to the proposed pairwise covariance properly weighted LDA (pcLDA). The kernel version of pcLDA is presented to handle nonlinear projections. Efficient algorithms are presented to efficiently compute the proposed models. Introduction In the big data era, a large number of high-dimensional data (i.e., DNA microarray, social blog, image scenes, etc) are available for data analysis in different applications. Linear Discriminant Analysis (LDA) (Hastie, Tibshirani, and Friedman 2001) is one of the most popular methods for dimension reduction, which has shown state-of-the-art performance. The key idea of LDA is to find an optimal linear transformation which projects data into a low-dimensional space, where the data achieves maximum inter-class separability. The optimal solution to LDA is generally achieved by solving an eigenvalue problem. Despite the popularity and effectiveness of LDA, however, in standard LDA model, instead of emphasizing the pairwise-class distances, it simply takes an average of metrics computed in different pairs (i.e., computation of between-class scatter matrix Sb or within-class scatter matrix Sw). Thus, some pairwise class distances are depressed, especially for those pairs whose original class distances are relatively large. To overcome this issue, in this paper, we present a new formulation for pairwise linear discriminant analysis. To obtain a discriminant projection, the proposed method considers all the pairwise between-class and with-class distances. We call it “pairwise-covariance LDA (pcLDA)”. Then, the Copyright c 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. pcLDA problem is cast into solving an optimization problem, which maximizes the class separability computed from pairwise distance. An efficient algorithm is proposed to solve the resultant problem, and experimental results indicate the good performance of the proposed method. A new perspective of LDA The standard linear discriminant analysis (LDA) is to seek a projection G = (g1, · · · ,gK 1) 2 <p⇥(K 1) which maximizes the class separability by solving,